Picking system, control device, picking method, and storage medium

ABSTRACT

According to one embodiment, a picking system includes a picking robot and a control device. The picking robot transfers an object from a first space to a second space by using a robot hand. The control device controls the picking robot. When a first measurement result related to a shape of the object in the first space when viewed along a first direction is acquired, the control device performs a first calculation of calculating a position candidate for placing the object in the second space based on the first measurement result. When a second measurement result related to a shape of the object when viewed along a second direction is acquired, the control device performs a second calculation of calculating a position of the robot hand when placing the object in the second space based on the second measurement result and the position candidate.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2021-184953, filed on Nov. 12, 2021; theentire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to a picking system, acontrol device, a picking method, and a storage medium.

BACKGROUND

There is a picking system that transfers an object. Picking systemtechnology that can reduce the time necessary for the picking task isdesirable.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view showing a picking system according to anexample;

FIG. 2 is a block diagram schematically showing the functionalconfiguration of the control device;

FIGS. 3A to 3C are schematic views for describing the method formeasuring the second measuring instrument;

FIG. 4 is a schematic view showing a processing procedure of the pickingsystem according to the embodiment;

FIG. 5 is a flowchart showing processing by the placement plangenerator;

FIG. 6 is a flowchart showing calculation processing of the positioncandidate calculator;

FIG. 7 is a flowchart showing the search method of the positioncandidate of the position candidate calculator;

FIG. 8 is a flowchart showing the general concept of the processing ofthe hand position calculator;

FIG. 9 is a flowchart showing processing of the hand positioncalculator; and

FIG. 10 is a schematic view showing a hardware configuration.

DETAILED DESCRIPTION

According to one embodiment, a picking system includes a picking robotand a control device. The picking robot transfers an object from a firstspace to a second space by using a robot hand. The control devicecontrols the picking robot. When a first measurement result related to ashape of the object in the first space when viewed along a firstdirection is acquired, the control device performs a first calculationof calculating a position candidate for placing the object in the secondspace based on the first measurement result. When a second measurementresult related to a shape of the object when viewed along a seconddirection in an action of the robot hand on the object is acquired, thecontrol device performs a second calculation of calculating a positionof the robot hand when placing the object in the second space based onthe second measurement result and the position candidate. The seconddirection crosses the first direction.

Various embodiments are described below with reference to theaccompanying drawings.

The drawings are schematic and conceptual; and the relationships betweenthe thickness and width of portions, the proportions of sizes amongportions, etc., are not necessarily the same as the actual values. Thedimensions and proportions may be illustrated differently amongdrawings, even for identical portions.

In the specification and drawings, components similar to those describedpreviously or illustrated in an antecedent drawing are marked with likereference numerals, and a detailed description is omitted asappropriate.

FIG. 1 is a schematic view showing a picking system according to anexample.

As shown in FIG. 1 , the picking system 1 according to the exampleincludes a picking robot 10, a first measuring instrument 21, a secondmeasuring instrument 22, a third measuring instrument 23, and a controldevice 30.

Herein, an X-direction, a Y-direction (a second direction), and aZ-direction (a first direction) are used in the description of theembodiments. The X-direction and the Y-direction cross each other. TheZ-direction crosses an X-Y plane (a first plane). For example, theX-direction and the Y-direction are parallel to a horizontal plane. TheZ-direction is parallel to a vertical direction.

The picking robot 10 transfers an object placed in a first space SP1inside a first container 41 to a second space SP2 inside a secondcontainer 42. More specifically, the first container 41 has a firstopening OP1 facing the Z-direction. The second container 42 has a secondopening OP2 facing the Z-direction. The picking robot 10 removes theobject from the first container 41 via the first opening OP1 and movesthe object into the second container 42 via the second opening OP2. Thepicking robot 10 includes a robot hand 11, a robot arm 12, and a housing13.

The robot hand 11 holds (stably grips) the object. For example, therobot hand 11 holds the object by one of suction-gripping, pinching, orjamming. In the example of FIG. 1 , the robot hand 11 includes multiplefingers 11 a. The multiple fingers 11 a hold the object by pinching theobject. The robot hand 11 is mounted to the robot arm 12.

The robot arm 12 moves the robot hand 11. In the example shown in FIG. 1, the robot arm 12 is a vertical articulated robot that has six degreesof freedom; and the robot hand 11 is mounted to the tip of the robot arm12. The robot arm 12 may be a horizontal articulated robot, a linearrobot, an orthogonal robot, or a parallel link robot. The robot arm 12may include a combination of at least two selected from a verticalarticulated robot, a horizontal articulated robot, a linear robot, anorthogonal robot, and a parallel link robot. The robot arm 12 is mountedto the housing 13.

The housing 13 supports the robot arm 12 and is fixed to the floorsurface. A power supply device for driving electric actuators such asmotors, a cylinder, tank, and compressor for driving fluid actuators,various safety mechanisms, etc., may be housed in the housing 13. Thecontrol device 30 may be housed in the housing 13.

The first measuring instrument 21 measures the shape when viewed alongthe Z-direction of the object placed in the first space SP1. Forexample, the first measuring instrument 21 includes an imaging part 21a. The imaging part 21 a is a camera including one or two selected froman image sensor and a distance sensor. The imaging part 21 a images theobject in the first space SP1 when viewed along the Z-direction andacquires an image (a still image). The imaging part 21 a may acquire avideo image and cut out a still image from the video image. The imagingpart 21 a transmits the image to the control device 30.

The control device 30 measures the shape of a first surface (the uppersurface), which crosses the Z-direction, of the object based on theimage. In the picking system 1, the imaging part 21 a and the controldevice 30 function as the first measuring instrument 21. The measurementresult (a first measurement result) of the first measuring instrument 21includes first shape information related to the shape of the firstsurface of each object. An image processing device other than thecontrol device 30 may be embedded in the imaging part 21 a and used asthe first measuring instrument 21.

The second measuring instrument 22 measures the shape when viewed alongthe Y-direction of the object while being acted on by the picking robot10. For example, the second measuring instrument 22 includes a lightcurtain 22 a. The light curtain 22 a includes a light projector 22 a 1and a light receiver 22 a 2. The light curtain 22 a includes a sensingregion SR facing the first opening OP1 when viewed along theZ-direction. The sensing region SR is a region that transmits the lightemitted from the light projector 22 a 1. The light curtain 22 a detectsthe object passing through the sensing region SR. The light curtain 22 atransmits the detection result to the control device 30.

The control device 30 measures the shape of a second surface (the sidesurface), which crosses the Y-direction, of the object based on theZ-direction position of the light curtain 22 a, the time at which theobject passed through the sensing region SR, the Z-direction position ofthe robot hand 11 at the time, etc. In the picking system 1, the lightcurtain 22 a and the control device function as the second measuringinstrument 22. The measurement result (a second measurement result) ofthe second measuring instrument 22 includes second shape informationrelated to the shape of the second surface. An arithmetic device otherthan the control device 30 may be included together with the lightcurtain 22 a and may be used as the second measuring instrument 22.

Instead of the light curtain 22 a, the second measuring instrument 22may include an imaging part or a distance sensor such as a laserrangefinder, etc. The control device 30 measures the Z-direction lengthof the object based on an image or the measurement result of thedistance sensor.

The third measuring instrument 23 measures the shape of the objectplaced in the second space SP2 when viewed along the Z-direction. Forexample, the third measuring instrument 23 includes an imaging part 23a. The imaging part 23 a is a camera including one or two selected froman image sensor and a distance sensor. The imaging part 23 a images thesecond space SP2 in the Z-direction and acquires an image (a stillimage). The imaging part 23 a may acquire a video image and cut out astill image from the video image. The imaging part 23 a transmits theimage to the control device 30.

The control device 30 measures the shape of the object placed in thesecond space SP2 based on the image. In the picking system 1, theimaging part 23 a and the control device 30 function as the thirdmeasuring instrument 23. The measurement result (a third measurementresult) of the third measuring instrument 23 includes obstacleinformation related to the three-dimensional shape of the object placedin the second space SP2. An image processing device other than thecontrol device 30 may be embedded in the imaging part 23 a and used asthe third measuring instrument 23.

In addition to the calculation described above, the control device 30controls the picking robot 10. For example, the control device 30 movesthe robot hand 11 and adjusts the posture of the robot hand 11 byoperating the drive axes of the robot arm 12. Also, the control device30 causes the robot hand 11 to hold the object and release the object.

FIG. 2 is a block diagram schematically showing the functionalconfiguration of the control device.

The control device 30 includes an integrating part 31, a measurementinformation processor 32, a holding plan generator 33, a placement plangenerator 34, an operation plan generator 35, and a robot control device36.

The integrating part 31 manages, implements, and causes the generationof task plans by the picking system 1 based on input information of theuser from an external interface (I/F) 37, the input of a pickinginstruction from a higher-level system, the state of the picking system1, etc.

The measurement information processor 32 controls the imaging part 21 a,the light curtain 22 a, and the imaging part 23 a. The measurementinformation processor 32 processes information obtained from the imagingpart 21 a, the light curtain 22 a, and the imaging part 23 a andgenerates a motion plan and information necessary for operation control,error detection, etc. The motion plan includes an operation plan thatrelates to an operation of the picking robot 10. The measurementinformation processor 32 performs a portion of the functions as thefirst to third measuring instruments 21 to 23.

For example, the measurement information processor 32 segments the imagethat is imaged by the imaging part 21 a and generates the first shapeinformation by using the result of the segmentation. In thesegmentation, the objects that are visible in the image are identified;and the image is subdivided into at least one region. Each regioncorresponds respectively to an object. The first shape information isrelated to the shape of the first surface of each object in the firstcontainer 41 and includes the segmentation result of the first surface,the X-direction length and the Y-direction length of the first surfaceof the object, the position of the first surface of the object in theX-Y plane, etc. The lengths and positions are calculated based on thesegmentation result. For example, the actual length of each object iscalculated based on the distance between the imaging part 21 a and theobject and the length (the number of pixels) in the X-direction or theY-direction of the object in the image. Similarly, the position of thefirst surface of each object in the X-Y plane is calculated based on theposition of the first surface in the image and the distance between theimaging part 21 a and the object.

The measurement information processor 32 generates the second shapeinformation based on the detection result of the light curtain 22 a. Thesecond shape information is related to the shape of at least a portionof the second surface of the object that is held. Specifically, thesecond shape information includes the Z-direction length (the height) ofthe at least a portion of the second surface.

FIGS. 3A to 3C are schematic views for describing the method formeasuring the second measuring instrument.

The method for measuring the Z-direction length of the object will bedescribed with reference to FIGS. 3A and 3B. The control device 30records the angles of the joints of the robot arm 12 at a prescribedinterval. Also, the light curtain 22 a detects the existence or absenceof a light-shielding object between the light projector 22 a 1 and thelight receiver 22 a 2 at a prescribed interval.

As shown in FIG. 3A, when the picking robot 10 holds an object, at leasta portion of light L emitted from the light projector 22 a 1 isobstructed by the robot hand 11 or the robot arm 12 and is not incidenton the light receiver 22 a 2. As the picking robot 10 raises the object,the light L becomes incident on the light receiver 22 a 2 when oneZ-direction end (the lower end) of the object passes through the sensingregion SR as shown in FIGS. 3B and 3C.

The light curtain 22 a records a second time t2 at which the light Lthat had been obstructed is first detected by the light receiver 22 a 2.The control device 30 calculates a first position z1 in the Z-directionof the robot hand 11 at a first time t1 directly before the second timet2 based on the angles of the joints of the robot arm 12 at the firsttime t1. For example, the position of the tool center point (TCP) of therobot hand 11 is calculated as the position of the robot hand 11. Thecontrol device 30 calculates a second position z2 in the Z-direction ofthe robot hand 11 at the second time t2 based on the angles of thejoints of the robot arm 12 at the second time t2. The control device 30estimates (z2+z1)/2 to be a position zH of the robot hand 11 when theobject passed through the sensing region SR of the light curtain 22 a.The control device 30 refers to a position zL in the Z-direction atwhich the light curtain 22 a is located. The position zL of the lightcurtain 22 a is preregistered. The control device 30 calculates zH−zL asa height SZ of the object.

As shown in FIG. 3C, the area measured by the second measuringinstrument 22 may not be the entire second surface of the object. It issufficient for the second measuring instrument 22 to be able to measurethe length (the distance) between the TCP and the lower end of theobject. In other words, the length is the protrusion amount of theobject from the tip of the robot hand 11. The length may be calculatedusing a reference other than the TCP. For example, when a control pointis set at the tip of the robot arm 12, the length is calculated usingthe control point as a reference. In the case of a point mechanicallyfixed with respect to a portion of the robot arm 12, the length can bemeasured using the point as a reference. Also, according to theconfiguration of the robot hand 11, the protrusion amount that ismeasured by the second measuring instrument 22 may be equal to theactual Z-direction length of the object. For example, when the robothand 11 holds only the upper surface of the object by suction-gripping,the Z-direction length of the entire object may be calculated as theprotrusion amount.

The measurement information processor 32 generates the obstacleinformation based on the image that is imaged by the imaging part 23 a.The obstacle information includes the position in the X-Y plane of eachobject in the second space SP2, the Z-direction position of the uppersurface of each object, etc.

The holding plan generator 33 generates a holding plan. The holding planincludes the holding method of the object, the holding position of therobot arm 12 when holding the object, the holding posture, the via-pointuntil reaching the holding position, etc.

The placement plan generator 34 generates a placement plan. Theplacement plan includes the placement position of the robot arm 12 whenreleasing the held object into the second container 42, the placementposture, the via-point until reaching the placement position, etc.

The operation plan generator 35 generates operation information of therobot arm 12. The operation information includes information related toa holding operation, a transfer operation, and a placement operation.The holding operation is the operation of the tip of the robot arm 12from above the holding position until reaching the holding position andthe holding posture. The transfer operation is the operation of the tipof the robot arm 12 from above the holding position until being abovethe placement position. The placement operation is the operation of thetip of the robot arm 12 from above the placement position until reachingthe placement position and the placement posture.

The robot control device 36 controls the picking system 1 including thepicking robot 10 according to the information generated by the holdingplan generator 33, the placement plan generator 34, or the operationplan generator 35, the operation switching instructions from theintegrating part 31, etc.

The external I/F 37 inputs and outputs data between the integrating part31 (the control device 30) and the external device (not illustrated).

FIG. 4 is a schematic view showing a processing procedure of the pickingsystem according to the embodiment.

The integrating part 31 receives a picking instruction from the externalI/F 37 (step S0). For example, the picking instruction is transmittedfrom a higher-level host computer. The integrating part 31 instructs themeasurement information processor 32 to image the first container 41.The measurement information processor 32 causes the imaging part 21 a toimage the interior of the first container 41 (step S1) and generates thefirst shape information. After the imaging of the first container 41,the holding plan generator 33 generates a holding plan (step S2). Inparallel, the measurement information processor 32 causes the imagingpart 23 a to image the interior of the second container 42 (step S3) andgenerates the obstacle information.

After the generation of the holding plan by the holding plan generator33 is completed, the robot control device 36 performs a holdingoperation based on the generated holding plan (step S4). In parallel,the placement plan generator 34 calculates a position candidate forplacing the object to be transferred in the second space SP2 based onthe holding plan and the imaging result of the second container 42 (stepS5). The placement plan generator 34 calculates the priority of theposition candidate (step S6). The placement plan generator 34 stores thecalculated position candidate and priority. After completing the holdingoperation, the robot control device 36 performs a transfer operation(step S7). In the transfer operation, the object that is held is liftedand transferred to the second container 42. In the transfer operation,the measurement information processor 32 causes the light curtain 22 ato detect the object that is held (step S8) and generates the secondshape information.

The placement plan generator 34 calculates the position of the robothand 11 when placing the object in the second container 42 based on thesecond shape information and the position candidate (step S9). Herein,the position of the robot hand 11 when placing the object in the secondcontainer 42 is called the “hand position”. After calculating the handposition, the robot control device 36 performs a placement operation(step S10). After completing the placement operation, it is determinedwhether or not the instructed quantity of objects have been transferred(step S11). Steps S1 to S10 are repeated until the instructed quantityof objects are transferred.

FIG. 5 is a flowchart showing processing by the placement plangenerator.

The placement plan generator 34 includes a position candidate calculator34 a and a hand position calculator 34 b. The position candidatecalculator 34 a starts processing when the generation of the holdingplan is completed. The position candidate calculator 34 a calculates theposition candidate inside the second container 42 of the object based onthe first shape information, the obstacle information, and the holdingplan (step S5). The position candidate is a candidate of the position ofthe transferred object when placed in the second container 42.Continuing, the position candidate calculator 34 a calculates thepriority of each position candidate (step S6). The position candidatecalculator 34 a stores the position candidate and the priority insidethe placement plan generator 34.

The hand position calculator 34 b starts processing when the measurementof the object by the second measuring instrument 22 is completed. Thehand position calculator 34 b calculates the hand position of the objectby using the position candidate calculated by the position candidatecalculator 34 a and the second shape information obtained by the secondmeasuring instrument 22 (step S9). Also, when calculating the handposition, the position of the robot hand 11 corresponding to the handposition is calculated. Subsequently, the operation plan generator 35generates operation information based on the calculated position of therobot hand 11. A placement operation is performed based on the operationinformation.

FIG. 6 is a flowchart showing calculation processing of the positioncandidate calculator.

The calculation processing (a first calculation, etc.) of the positioncandidate calculator 34 a for calculating the position candidate will bedescribed with reference to FIG. 6 . First, the first shape informationof the transferred object is acquired (step S51). More specifically, thesegmentation result and the size in the X-Y plane included in the firstshape information are acquired. Plane mesh data is generated using thesegmentation result (step S52). The mesh data is generated bypartitioning a portion of the image subdivided by the segmentation intoa lattice shape. The generated plane mesh data is stored as “MESH_OBJ”.The plane mesh data indicates the shape in the X-Y plane of the firstsurface of the transferred object.

The obstacle information is acquired (step S53). Three-dimensional meshdata that indicates the shape of the obstacle in the second space SP2 isgenerated using the obstacle information (step S54). The generatedthree-dimensional mesh data is stored as “MESH_TOTE”.

The candidate of the position at which the held object will be placed issearched using the plane mesh data and the three-dimensional mesh data(step S56). For example, a grid search, a binary search tree, or a MonteCarlo Tree Search (MCTS) is used as the search method. Normally,multiple position candidates are obtained unless many objects are to beplaced in the second container 42, the second container 42 isexcessively small, etc. Favorably, all positions at which placement ispossible are calculated as the position candidates. To reduce thecalculation time of the position candidates, the number of positioncandidates to be calculated may be pre-specified. In such a case, theposition candidate calculator 34 a ends the search when the specifiednumber of position candidates are calculated.

The priority of each position candidate is calculated (step S6). Theposition candidate calculator 34 a stores the position candidate and thepriority.

FIG. 7 is a flowchart showing the search method of the positioncandidate of the position candidate calculator.

In FIG. 7 , X0 is the origin coordinate of the second container 42 inthe X-direction. Y0 is the origin coordinate of the second container 42in the Y-direction. Z0 is the origin coordinate of the second container42 in the Z-direction. For example, the Z-direction position of thebottom surface of the second container 42 is set as Z0. For example, oneof the four corners of the second container 42 is set as the originposition in the X-Y plane. The bottom surface of the second container 42is set as the origin position in the Z-direction. SX is the length ofthe second container 42 in the X-direction. SY is the length of thesecond container 42 in the Y-direction. SZ is the length of the secondcontainer 42 in the Z-direction. SX, SY, and SZ are preset.

First, X0 is substituted in the variable X (step S56 a). It isdetermined whether or not the variable X is greater than X0+SX (step S56b). In other words, it is determined whether or not the searchedX-coordinate is positioned outside the second container 42. When thevariable X is greater than X0+SX, the search ends. When the variable Xis not greater than X0+SX, Y0 is substituted in the variable Y (step S56c). It is determined whether or not the variable Y is greater than Y0+SY(step S56 d). In other words, it is determined whether or not thesearched Y-coordinate is positioned outside the second container 42.When the variable Y is greater than Y0+SY, the value of ΔX added to thecurrent variable X is substituted in the variable X (step S56 e). StepS56 b is re-performed. In other words, the searched X-coordinate isslightly shifted in the X-direction.

When the variable Y is not greater than Y0+SY in step S56 d, Z0+SZ issubstituted in the variable Z (step S56 f). The variable X, the variableY, and the variable Z at the timing of the completion of step S56 f areset as the MESH_OBJ coordinate (X, Y, Z) (step S56 g). MESH_OBJ is theplane mesh data of the transferred object. It is determined whether ornot the MESH_OBJ coordinate (X, Y, Z) crosses MESH_TOTE (step S56 h).MESH_TOTE is the three-dimensional mesh data inside the second container42. In step S56 h, it is determined whether or not the bottom surface ofthe object will contact an obstacle (another object or the bottomsurface or side surface of the second container 42) when placing theobject at the coordinate (X, Y, Z).

When the coordinate (X, Y, Z) does not cross MESH_TOTE, the value of ΔZsubtracted from the current variable Z is substituted in the variable Z(step S56 i). Step S56 g is re-performed. In other words, the loweringof the Z-direction position is repeated until the bottom surface of theplaced object contacts an obstacle. When the coordinate (X, Y, Z)crosses MESH_TOTE, that coordinate (X, Y, Z) is stored as a positioncandidate (step S56 j). The priority of the stored position candidate iscalculated (step S6). When the priority is calculated, the value of ΔYadded to the current variable Y is substituted in the variable Y (stepS56 k). Subsequently, step S56 d is re-performed.

The method for calculating the priority can be arbitrarily set accordingto the placement reference that is considered important. As an example,the objects are preferentially placed from the bottom surface or cornersof the second container 42. In such a case, a score Sc that indicatesthe priority is calculated by the following Formula 1. In the formula,a, b, and c are weighting factors. X, Y, and Z are coordinates of theposition candidate respectively in the X-direction, the Y-direction, andthe Z-direction. X0 and Y0 are the coordinates in the X-Y plane of thepreferential placement corner. X1 and Y1 are coordinates in the X-Yplane of the corner positioned diagonal to the corner at the coordinate(X0, Y0). The priority increases as the score Sc increases.

Sc=a(1-|X-X0|/|X1−X0|)+b(1-|Y-Y0|Y1−Y0|)+c(1-|Z-Z0|/|Z1-Z0|)  [Formula1]

In the example shown in FIG. 7 , the priority is calculated each timethe position candidate is calculated. The priorities may be calculatedfor multiple position candidates after the position candidates arecalculated.

FIG. 8 is a flowchart showing the general concept of the processing ofthe hand position calculator.

The calculation processing (a second calculation, etc.) of the handposition calculator 34 b for calculating the hand position will bedescribed with reference to FIG. 8 . First, the second shape informationis acquired, and the protrusion amount (SZ_OBJ) in the Z-direction ofthe object is set (step S90). The three-dimensional mesh data“MESH_TOTE” that indicates the obstacle information inside the secondcontainer 42 is acquired (step S91). As MESH_TOTE, the three-dimensionalmesh data that is generated by the position candidate calculator 34 amay be utilized, or three-dimensional mesh data may be newly generated.

The actual holding position of the object by the robot hand 11 isacquired from the holding result (step S92). The shape of the hand isacquired as mesh data “MESH_HAND” (step S93). The acquired mesh data isreferenced to the actual holding position of the object by the robothand 11. For example, the mesh data “MESH_HAND” is prepared beforehand.The mesh data “MESH_HAND” may be generated based on the image acquiredby the imaging part 21 a. The hand position is determined using the meshdata “MESH_HAND”, the three-dimensional mesh data “MESH_TOTE”, SZ_OBJ,and the position candidate calculated by the position candidatecalculator 34 a (step S94).

FIG. 9 is a flowchart showing processing of the hand positioncalculator.

Details of step S94 of the flowchart shown in FIG. 8 will be describedwith reference to FIG. 9 . First, one position candidate among at leastone position candidate calculated by the position candidate calculator34 a is extracted (step S94 a). In step S94 a, the position candidate isextracted in order of decreasing priority. The extracted positioncandidate is set as the coordinate (X_OBJ, Y_OBJ, Z_OBJ) of the bottomsurface of the transferred object (step S94 b).

An example of the specific method for setting the coordinate (X_OBJ,Y_OBJ, Z_OBJ) will now be described. The actual holding position of therobot hand 11 in the X-Y plane acquired in step S92 is taken as (X_GTCP,Y_GTCP). The coordinate of the object in the first container 41 is takenas (X_GOBJ, Y_GOBJ). In such a case, the relative position of theholding position of the robot hand 11 and the position of the object is(X_REL, Y_REL)=(X_GOBJ-X_GTCP, Y_GOBJ−Y_GTCP). When the positioncandidate of the object is (X_C, Y_C, Z_C), the hand position candidatefor placement considering the relative position is (X_OBJ, Y_OBJ,Z_OBJ)=(X_C−X_REL, Y_C−Y_REL, Z_C).

The value of the protrusion amount SZ_OBJ in the Z-direction of theobject added to the Z-coordinate Z_OBJ of the bottom surface is set as“Z_TCP” (step S94 c). Z_TCP is the Z-direction position of the TCP ofthe robot hand 11. It is determined whether or not MESH_HAND crossesMESH_TOTE when MESH_HAND is placed on the coordinate (X_OBJ, Y_OBJ,Z_TCP) (step S94 d).

When MESH_HAND crosses MESH_TOTE, the value of ΔZ added to Z_TCP is setas the new Z_TCP (step S94 e). It is determined whether or not the raiseamount of the new Z_TCP for Z_TCP set in step S94 c is greater than athreshold (step S94 f). The object falls from a higher position as theaddition of ΔZ is repeated. A height from which the object can fallwithout damage is set as the threshold. When the raise amount is notgreater than the threshold, step S94 d is re-performed using the newZ_TCP.

When MESH_HAND does not cross MESH_TOTE in step S94 d, the angles of thejoints of the robot arm 12 corresponding to Z_TCP are calculated byinverse kinematics (step S94 g). It is determined whether or not theangles of the joints are within the range of movement (step S94 h). Whenthe angles of the joints are within the range of movement, (X_OBJ,Y_OBJ, Z_TCP) is determined as the hand position (step S94 i), and theselect processing ends.

When the raise amount is greater than the threshold in step S94 f orwhen the angles of the joints are outside the range of movement in stepS94 h, it is determined whether or not an unextracted position candidateexists (step S94 j). When an unextracted position candidate exists, theprocessing is re-performed for another position candidate. When anotherposition candidate does not exist, the calculation processing of thehand position ends. This means that the object cannot be placed in thesecond container 42.

Advantages of the embodiment will now be described.

In the picking task of a robot, it is desirable to reduce impacts whentransferring so that the object is not damaged. To reduce impacts whentransferring, it is favorable to acquire the size (the three-dimensionalshape) of the object in the X-direction, the Y-direction, and theZ-direction. Based on the acquired three-dimensional shape, the objectcan be placed in the second space SP2 of the transfer destinationwithout colliding with surrounding obstacles. In particular, byaccurately acquiring the size in the Z-direction of the object, contactwith obstacles when transferring the object or falling when the objectis released can be prevented, and the impacts to the object can bereduced.

When multiple objects are placed in the first container 41 of thetransfer origin, it is difficult to acquire accurate three-dimensionalshapes based on the measurement result of the first measuring instrument21. For example, the Z-direction length of one object cannot be measuredwhen a portion of the one object is hidden by another object. A methodfor acquiring the three-dimensional shape of the transferred object maybe considered in which the object is measured after the robot hand 11acts on the object. The three-dimensional shape of the object can beacquired by the action of the robot hand 11 exposing the hidden portionof the object. On the other hand, the calculation of the hand positionby using the three-dimensional shape is computation-intensive andrequires time. If the calculation time after the action of the robothand 11 on the object is long, it is necessary to stop the picking robot10 until the calculation result is obtained. Therefore, the time of thepicking task lengthens, and the work efficiency decreases.

In the picking system 1 according to the embodiment, the firstcalculation of calculating the position candidate of the transferredobject is performed when the first measurement result of the firstmeasuring instrument 21 is obtained. The position candidate iscalculated based on the first measurement result from the firstmeasuring instrument 21 and is a candidate of the position of the objectin the second space SP2. As described above, it is difficult for thefirst measuring instrument 21 to accurately measure the Z-directionlength of the transferred object. However, even when the placementposition of the final object cannot be calculated, the first measurementresult makes it possible to calculate candidates of positions at whichthe object can be placed. In other words, the first calculation can bestarted before acquiring the second measurement result from the secondmeasuring instrument 22. Continuing, when the second measurement resultis obtained in the picking system 1, the position of the robot hand 11when placing the object in the second space SP2 is calculated based onthe second measurement result and the position candidate. In otherwords, the second calculation is started after the first calculation andafter acquiring the second measurement result. In the secondcalculation, the hand position can be quickly calculated because theposition candidates that are used are already calculated.

According to the picking system 1, compared to when the calculations ofthe hand position and the placement position of the object are startedafter the three-dimensional shape is obtained, the hand position can becalculated at an earlier timing. Therefore, the time that the pickingrobot 10 is stopped while calculating the hand position can be reduced.For example, the hand position can be calculated without stopping thepicking robot 10. As a result, the time of the picking task can bereduced, and the work efficiency can be increased. Because the handposition can be calculated based on the three-dimensional shape of theobject, the contact of the object with obstacles when transferring orthe object dropping when released can be prevented, and impacts to theobject can be reduced. Because the three-dimensional shape of the objectis measured by the first and second measuring instruments 21 and 22, itis unnecessary to prepare a three-dimensional model of the object, etc.,beforehand.

According to the embodiment, the time of the picking task can be reducedwhile reducing impacts to the object when transferring.

An example is described above in which the Z-direction is parallel tothe vertical direction; and the X-direction and the Y-direction areparallel to a horizontal plane. Embodiments are not limited to such anexample. For example, the Y-direction may be parallel to the verticaldirection; and the X-direction and the Z-direction may be parallel to ahorizontal plane. In any case, the shape of the object is measured fromdifferent directions by the first and second measuring instruments 21and 22. The first calculation is performed after the measurement by thefirst measuring instrument 21 and before the measurement by the secondmeasuring instrument 22; and the second calculation is performed afterthe measurement by the second measuring instrument 22. Thereby, the timeof the picking task can be reduced while reducing impacts to the objectwhen transferring.

FIG. 10 is a schematic view showing a hardware configuration.

The control device 30 includes, for example, the hardware configurationshown in FIG. 10 . A processing device 90 shown in FIG. 10 includes aCPU 91, ROM 92, RAM 93, a memory device 94, an input interface 95, anoutput interface 96, and a communication interface 97.

The ROM 92 stores programs that control the operations of a computer.Programs that are necessary for causing the computer to realize theprocessing described above are stored in the ROM 92. The RAM 93functions as a memory region into which the programs stored in the ROM92 are loaded.

The CPU 91 includes a processing circuit. The CPU 91 uses the RAM 93 aswork memory to execute the programs stored in at least one of the ROM 92or the memory device 94. When executing the programs, the CPU 91executes various processing by controlling configurations via a systembus 98.

The memory device 94 stores data necessary for executing the programsand/or data obtained by executing the programs.

The input interface (I/F) 95 connects the processing device 90 and aninput device 95 a. The input I/F 95 is, for example, a serial businterface such as USB, etc. The CPU 91 can read various data from theinput device 95 a via the input I/F 95.

The output interface (I/F) 96 connects the processing device 90 and anoutput device 96 a. The output I/F 96 is, for example, an image outputinterface such as Digital Visual Interface (DVI), High-DefinitionMultimedia Interface (HDMI (registered trademark)), etc. The CPU 91 cantransmit data to the output device 96 a via the output I/F 96 and causethe output device 96 a to display an image.

The communication interface (I/F) 97 connects the processing device 90and a server 97 a outside the processing device 90. The communicationI/F 97 is, for example, a network card such as a LAN card, etc. The CPU91 can read various data from the server 97 a via the communication I/F97. The images from the imaging parts 21 a and 23 a and the detectionresult from the light curtain 22 a are stored in the server 97 a.

The memory device 94 includes at least one selected from a hard diskdrive (HDD) and a solid state drive (SSD). The input device 95 aincludes at least one selected from a mouse, a keyboard, a microphone(audio input), and a touchpad. The output device 96 a includes at leastone selected from a monitor and a projector. A device such as a touchpanel that functions as both the input device 95 a and the output device96 a may be used.

The processing of the various data described above may be recorded, as aprogram that can be executed by a computer, in a magnetic disk (aflexible disk, a hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW,DVD-ROM, DVD±R, DVD±RW, etc.), semiconductor memory, or anothernon-transitory computer-readable storage medium.

For example, the information that is recorded in the recording mediumcan be read by the computer (or an embedded system). The recordingformat (the storage format) of the recording medium is arbitrary. Forexample, the computer reads the program from the recording medium andcauses a CPU to execute the instructions recited in the program based onthe program. In the computer, the acquisition (or the reading) of theprogram may be performed via a network.

According to the embodiments described above, a picking system, acontrol device, a picking method, a program, and a storage medium areprovided in which the time of the picking task can be reduced whilereducing impacts to the object when transferring.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the invention. The above embodiments can be practiced incombination with each other.

What is claimed is:
 1. A picking system, comprising: a picking robottransferring an object from a first space to a second space by using arobot hand; and a control device controlling the picking robot, whenacquiring a first measurement result related to a shape of the object inthe first space when viewed along a first direction, the control deviceperforming a first calculation of calculating a position candidate forplacing the object in the second space based on the first measurementresult, when acquiring a second measurement result related to a shape ofthe object when viewed along a second direction in an action of therobot hand on the object, the control device performing a secondcalculation of calculating a position of the robot hand when placing theobject in the second space based on the second measurement result andthe position candidate, the second direction crossing the firstdirection.
 2. The system according to claim 1, further comprising: afirst measuring instrument acquiring the first measurement result bymeasuring the shape of the object when viewed along the first direction;and a second measuring instrument acquiring the second measurementresult by measuring the shape of the object when viewed along the seconddirection.
 3. The system according to claim 2, wherein the secondmeasuring instrument measures the shape of the object while the objectis held by the picking robot.
 4. The system according to claim 2,wherein the first measuring instrument measures a shape of a firstsurface of the object, and the first surface crosses the firstdirection.
 5. The system according to claim 4, wherein the firstmeasuring instrument measures the shape of the object of the firstsurface by segmenting a region of the object based on an image of thefirst space.
 6. The system according to claim 2, wherein the secondmeasuring instrument measures a protrusion amount in the first directionof the object from a tip of the robot hand.
 7. The system according toclaim 6, wherein the second measuring instrument includes a sensingregion spreading along a first plane crossing the first direction, andthe second measuring instrument measures the protrusion amount based ona time at which the object passes through the sensing region while theobject is moved in the first direction by the picking robot.
 8. Thesystem according to claim 7, wherein the second measuring instrumentmeasures the protrusion amount based on a position in the firstdirection of the sensing region and a position in the first direction ofthe robot hand at the time.
 9. The system according to claim 1, furthercomprising: a third measuring instrument measuring a shape of the secondspace, in the first calculation, the control device calculating theposition candidate based on the first measurement result and a thirdmeasurement result from the third measuring instrument.
 10. The systemaccording to claim 9, wherein the third measuring instrument measures ashape of an obstacle in the second space, and the control device:calculates the position candidate in the first calculation based on theshape of the object and the shape of the obstacle; and calculates theposition of the robot hand in the second calculation based on the shapeof the object, the shape of the obstacle, and the position candidate.11. The system according to claim 1, wherein the control device:calculates a plurality of the position candidates in the firstcalculation; and calculates the position of the robot hand in the secondcalculation by using one of the plurality of position candidates and aprotrusion amount in the first direction of the object indicated by thesecond measurement result.
 12. The system according to claim 1, whereinthe first direction is parallel to a vertical direction, and the seconddirection is parallel to a horizontal direction.
 13. A control devicecontrolling a picking robot, the picking robot transferring an objectfrom a first space to a second space by using a robot hand, whenacquiring a first measurement result related to a shape of the object inthe first space when viewed along a first direction, the control deviceperforming a first calculation of calculating a position candidate forplacing the object in the second space based on the first measurementresult, when acquiring a second measurement result related to a shape ofthe object when viewed along a second direction in an action of therobot hand on the object, the control device performing a secondcalculation of calculating a position of the robot hand when placing theobject in the second space based on the second measurement result andthe position candidate, the second direction crossing the firstdirection.
 14. A picking method using a picking robot, the picking robottransferring an object from a first space to a second space by using arobot hand, the picking method comprising: when acquiring a firstmeasurement result related to a shape of the object in the first spacewhen viewed along a first direction, performing a first calculation ofcalculating a position candidate for placing the object in the secondspace based on the first measurement result; and when acquiring a secondmeasurement result related to a shape of the object when viewed along asecond direction in an action of the robot hand on the object,performing a second calculation of calculating a position of the robothand when placing the object in the second space based on the secondmeasurement result and the position candidate, the second directioncrossing the first direction.
 15. A non-transitory computer-readablestorage medium storing a program, the program causing a computer tocontrol a picking robot, the picking robot transferring an object from afirst space to a second space by using a robot hand, the program causingthe computer to perform: when a first measurement result related to ashape of the object in the first space when viewed along a firstdirection is acquired, a first calculation of calculating a positioncandidate for placing the object in the second space based on the firstmeasurement result; and when a second measurement result related to ashape of the object when viewed along a second direction in an action ofthe robot hand on the object is acquired, a second calculation ofcalculating a position of the robot hand when placing the object in thesecond space based on the second measurement result and the positioncandidate, the second direction crossing the first direction.